Index hyper geometric integral transform 

Yury A.Neretii^I 



This is a brief overview of the index hypergeometric transform (other terms for 
this integral operator are: Olevskii transform, Jacobi transform, generalized Mehler- 
Fock transform). We discuss applications of this transform to special functions and 
harmonic analysis. The text is an addendum to the Russian edition of the book by 
G. E. Andrews, R. Askey, and R. Roy, Special Functions, Encycl. of Math. Appl. 71, 
Cambridge Univ. Press, 1999. 



As is well-known, the continuous analog of Fourier series is the Fourier trans- 
form. It turns out that expansions in the Jacobi polynomials also have a continu- 
ous analog, namely the integral 'Jacobi transform' (the terminology is discussed 
below). The theory of this transform is rich, but there is no detailed modern 
exposition of this topic in the existing literature, this text also has not such a 
purpose. Numerous additional facts are contained in works [T], [3J, [B], |13|-|15j. 

E2) ED- US)- 

There is a well-known Askey- Wilson hierarchy of hypergeometric orthogonal 
polynomials (see [H]). There is a parallel hierarchy of hypegeometric integral 
transforms, see [2], [S], [3J, [12], [2D], [24]-[25], [34], [35], on multidimensional 
analogs, see [9J, [18], pQ. Our topic is neither the simplest object of this hierarchy 
(certainly the Hankel transform and the Kontorovich-Lebedev transform are 
simpler), nor the most complicated (see, for instance [8]). It is sufficiently 
simple to be versatile tool for special functions (see below Section 2), on the 
other hand it controls harmonic analysis on hyperbolic symmetric spaces (i.e., 
Lobachevsky spaces and their complex and quaternionian analogs), we briefly 
discuss this in Section 4, for more details see [2], [2"2"] . 

1 The index hypergeometric transform 

1.1. The Jacobi polynomials. Consider the Jacobi orthogonal system on 
the segment [0, 1], 



n . Hx) . '- i ); r i"t/ 3 , +1) 2Fl 



T(p + l)n\ 



-n, n + a + j3 + 1 
j8 + l ' 



We have 



7n:=II^H 2 = 

= / y^{xfx^{l-x) a dx 



T(n + a + l)T(n + j3 + 1) 
(2n + a + p + 1) n\ T(n + a + j3 + 1) ' 



(1.1) 
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For a function f(x) consider the numbers ('the Fourier coefficients') defined by 
the formula 



Cn(f) ■= / f(x)9Z-P(x)x a {l-xfdx. 

Jo 

Then the function f(x) can be restored by the formula 

n=0 < n 

Moreover, the following 'Plancherel formula' holds: 

f(x) g(x) - x) a dx = Y, — C M) c n (g). 



(1.2) 



/ 

Jo 



n=0 7 " 



The expansion in the Jacobi polynomials has a continuous analog, in which 
series are replaced by integrals. 

1.2. The index hypergeometric transform. Let b, c > 0. For a function 
/ on the half-line [0, oo) we define a function of the variable s ^ by 



JbJ{s) = [f] b ,c{s) = 
1 



r(b + c) Jo 



fix) 2 F 1 



b + is, b — is 
b + c ' 



J>+c-l( 



(l + x)"- c dx. (1.3) 



Theorem 1.1 a) The operator Jb, c is a unitary operator 

T(b + is)T(c + is) 2 



J b . c : i 2 ([0,oo), x^-^l + xf^dx^ -> L 2 ([0,oo), 



In other words, the following Plancherel formula holds 



T(2is) 



ds 



)• 



f 1 (x)f 2 (x)x b+c -\l+ X ) b - c dx 



b-c 



[/kc(*)[/k» 



T{b + is)T(c + is) 



T(2is) 



ds. 



b) The inverse operator is given by the formula 



fix) 



n 



& + is, b — is 
b + c ' 



r(6 + is)r( c + is) 



r(2is) 



ds. 
(1.4) 



Notice that the statement b) follows from a), because for a unitary operator 
[/ we have t/" 1 = [/*. 

As in the case of the Fourier transform, we have a question about a precise 
definition. For instance, we can say that the integral transform Jb. c is defined 
on functions with compact support, next we extend it by continuity to a unitary 
operator defined in the space L 2 . 



2 



1.3. Holomorphic extension to a strip. 
Lemma 1.2 Let f be integrable on R+ and 

f(x) = o{x~ a ~ e ), x -> +00, 

where e > 0. Then [f(s)]b_ c is holomorphic in the strip 

I Im s\ < a — b 

and satisfy the condition /(— s) = /(s) m i/ie s£up. 

Proof. This follows from the asymptotics for the hypergeometric function, 
(see g], V. 1, (2.3.2.9)) as x -> +00: 

2 Fi(6+is,6-w;6+^-x) = Axaj-^'+Aaaf^'+Ofx-^'-^+OCaj- 6- *"- 1 ) 

where 2js ^ Z, and Ai, A2 are certain constansts (for 2is € Z there arises an 
additional factor In x at the leading term) . □ 

1.4. The operator calculus. Denote by D the hypergeometric differential 
operator 

D := -x(x + 1)^ " [(c + 6) + (26 + l)a:] + b 2 (1.5) 

(in comparision with the common notation we replaced x by —2;). The hy- 
pergeometric functions in (|1.3p are (generalized) eigenfunctions of the operator 
D: 



D 2 F X 



~b + is, b — is 
b + c '~ X . 



= s 2 2F1 



b + is, b — is 
b+c '~ X 



(1.6) 



It is easy to see that D is formally self-adjoint in the following sense 

Df 1 (x)-Mx)x b+c -\l + x) b - c dx= / f 1 (x)-Df 2 (x)x b+c - 1 (l + x) b - c dx 



where fx, f 2 are smooth compactly supported functions on (0,oo) (in fact D is 
essentially self-adjoint, see below) and Theorem 11.11 is a theorem about expan- 
sion of D in eigenfunctions. 

Theorem 1.3 Let f, Df e L 2 , then 

[Df(s)] b , c = -s 2 f(s). (1.7) 

Proof. This is a rephrasing of formula (|1.6I) . 

Theorem 1.4 Let a function f be continuous on M + and satisfies the condition 

f{x)^o{ X - b - 1 - e )- 1 x^+00. (1.8) 

Then 

[xW)] b ,c = P[f(x)]b,c, (1.9) 
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where the difference operator Pg is given by 

, , (b — is)(c — is) . ... (b + is)(c + is) , , ., ... 

P9 (S) = (-2»)(1 - 2») (5(S + i} ^ 5(S)) + (2»)(l + 2») (5(S " l) ~ 5(S)) - 

(1.10) 

Remark. Emphasize amusing characteristics of this theorem. 

1. The operator P is a difference operator, but a shift s i— > s + i is in the 
imaginary direction, and integration is along the real axis. 

2. The transformation J^ 1 send the operator P to the operator of multiplica- 
tion by a function x, i.e., our operator J^ 1 determines a spectral decomposition 
of the difference operator P. For several examples of spectral decompositions 
of difference operators in imaginary direction, see [26] , [8] . 

3. The operator P is similar to difference operators, related to Wilson, 
continuous Hahn, continuous dual Hahn, Meixner-Pollachek orthogonal poly- 
nomials, see [AAR] . formulas (6.10.6), (6.10.9), (6.10.12) and Problem 6.37.C 
(see also [11]). The rational coefficients of the operator P are 'catenated' with 
the T-factors in the formula (11.41) . 

Proof. This is reduced to a verification of the identity 



P2F1 



is, b - 
b + c 



x 2F1 



is, b - 
b + c 



Theorem 1.5 Let f and f be continuous and decrease as (|1.8|) . Then 

[x(x + l)-^f] btC = H[f} hc , (1.11) 
where the difference operator H is given by 

tt / \ (b — is)(b+l — is)(c — is) , , ,. ... 

(b + is)(b + 1 + is)(c + is) , , , , .. . , . 

+ feiw^k {9{s - l) - 9{s)) - {b + c)9{s) - (L12) 

Proof. We evaluate Jf, )C -image of the commutator [x, D]. 

1.5. Historical remarks. The transformation Ji/2,1/2 w as introduced by 
F. G. Mehler [19] in 1881. He presented the inversion formula without proof (it 
has to be said that the formula is not obvious at all). A proof was published 
by V. A. Fock [7] in 1943. As a result, the transformation Jx/2,1/2 is called 
the Mehler-Fock transform. The general transformation Jfc. c was introduced 
by H.Weyl in 1910 the work 33 on the spectral theory of of differntial opera- 
tors. It seems that this result have not met the eye. Again this transform had 
appeared in the book of Titchmarsh [31 in 1946. In 1949 the transform was 
rediscovered by M. N. Olevsky [57], apparently this was related to his works on 
multi-dimensional Lobachevsky spaces. 

The most common terms for Jf, jC are the Olevsky transform and the Jacobi 
transform (introduced by T. Koornwinder) . 
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2 Applications to special functions 

Our first aim is to evaluate index hypergeometric transforms of some functions. 
We do this in Subsection 2.2 by the Mellin transform. Next, in 2.3-2.4 we 
demonstrate effectivity of the index transform as a tool of theory of special 
functions. 

2.1. The Mellin transform. Let fix) be a function defined on the half- 
line x > 0. Its Mellin transform is defined by the formula 



The domain of absolute convergence of this integral is a certain vertical strip 
of the form u < Res < v, the function F(s) is holomorphic in this strip, the 
boundaries of the strip can belong or do not belong the domain of convergence; 
a strip can be degenerated to a vertical line of the form Re s = u. Certainly, it 
can be empty. 

The Mellin transform is a unitary operator from L 2 (R, dx / x) to L 2 on ver- 
tical line Res = 1/2. In particular, 



Recall the theorem about convolution. If the domains of definition of F(s) = 
M/(s) and G(s) — M.g(s) have an intersection (a strip or a line), then M send 
the multiplicative convolution 



to F(s)G(s) (on the common domain of definition). 

Certainly, the Mellin transform is reduced to the Fourier transform by a 
substitution x = e y . From the point of view of abstract theory there is no 
difference between the Mellin transform and the Fourier transform. But their 
role in theory of special functions is different. 

2.2. A game in the Mellin transform. A short table of index 
transforms. Since we will meet long products of T-functions, we will use the 
following notation 



It can be represented as a linear combination of hypergeometric functions p F q 
with r-coefficients. The idea is explained in the book | A ARj . Section 2.4. A 






Tai, flfcl , = r(ai) . ..T(a k ) 

K k\- r(6i)...r(&o ' 

Consider arbitrary convergent Barnes-type integral 
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calculation requires a watching of some asymptotics, but it can be done once 
and forever in 'general case'. The final rules can be found in [30 , [28 . 

On the other hand there are unexpectedly many cases when the integral 
admits a simpler expression than the result of the general algorithm, see the 
tables of Prudnikov, Brychkov, Marichev, v. 3, [25], Chapter 8 (I do not know 
rational explanations of this phenomenon). 

Now we evaluate two auxiliary integrals. 

Lemma 2.1 



o {x + z)p 

z a -P 



>F 1 (p,q;r;-x)dx 



2ni 



r 


pioo 


r 


P, Q, P_ 


J —lOO 



t + a, p — t — a,p + t,q + t, —t 
r + t 



z l dt. (2.1) 



JO 



2m 



1 %F\ 


P,Q 

: —ujx 






r 






ioo 


r, w 












u,v,p, q 





u, v 
w ' 



-UJX 



dx 



a + t,u + t,v + t,p — a — t,q — a — t,—t 
r — a — t.w + t 



(2.2) 



Proof. To be dehnite, we evaluate the first integral. The Mellin transform 
of the function f(x) := x a ~ x j{x + z)p is B(s + a, p - s - a)z s+a ~ p . The Mellin 
transform of g{x) := 2F1 (p, q; r; —x) is evaluated in | A AR) . Section 2.4, and is 
a product of T-functions. Our integral is a convolution of xf(l/x) and g(x). 
Next, we observe that the Mellin transform of the function xf(l/x) is F(l — s). 
It remains to apply the theorem about convolution. □ 

Thus, our calculations was reduced to an rearrangement of T-functions. 

In the right hand side there are Barnes integrals, which can be represented 
as linear combinations of functions 3^2 and 4^3 respectively. We will not write 
them, instead of this we notice that for some values of the parameters T-factors 
in the right hand sides can cancel. 



Lemma 2.2 The transform Jb :C send 

-a- c , r(c + is)r(c- is) 



(1 



(1+x) 



b—a 



(x + z) 



c+b 



r(c + a)r(c + 6) ' 

p c + is, c — is 
c + a, c + b 

r(— u + 6) r(u + is)T(u — is) 



2^1 



c + is, c - 
c + a 



is 



T(a- 



T(b + is)T{b-is) 



(2.3) 
(2.4) 
(2.5) 
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2-Fl 



p + b,q + b 
a + b ' 



a + b 
p + q,p + b,q + b 



(l+x 

•r 



,b-a 



p + is,p — is,q + is, q — is 
a + is, a — is 



1F1 



p + is,p — is 

p + q ' 



(2 



2 Ft 



p + b,q + b 
a + b 



]-X 



(1 + 

a - 

q,p 



\b-o 



b 

b,q + b 



is,p — is,q ■ 
a + is, a 



■is,q 
is 



(2.7) 



2F1 



a + c, a + d _ 
a + b + c + d'~ 

T(a + b + c + d) ■ T(c + is)T(c - is)T(d + is)T(d - is) 
> r(a + c)r(a + d)T(b + c)T(b + d)T(c + d) ' 



(2.8) 



Proof. We look to the right hand side of (|2.2|) . If a = r, then two T-factors 
cancel. The rest is the integral representation of 2^1- This gives the second 
formula. Substituting z = 1 we get the first formula. 

Next, if z = 1, r = p + q-\- p, then we get one of Barnes integrals in the right 
hand side (see Theorem 2.4.3 of [AAR] 1. This gives us (12.51) . 

Further, we watch possible simplifications in the right hand side of (|2.2|l . 
After the substitution a — w — r we get a cancelation of four T-factors. This 
gives formula (|2.6I) . Substiting y = 1 to (|2.6[) we get (12.71) . 

To verify (|2.8j) . we observe that two T-factors in the right hand side of ()2.2[) 
cancel, and we again apply Theorem 2.4.3 of |A AR] . 

2.3. Game in the Plancherel formula. Thus we wrote a short table 
with 6 row for the transform Jb iC . Applying the Plancherel formula for J\> iC we 
can get an amusing collection of integrals. We present several examples. 

a) The De Branges-Wilson integral. Applying the Plancherel formula for 
pair of functions (1 + x)~ a ~ c and (1 + x)~ a ~ d , we after a trivial calculation we 
get the De Branges-Wilson integral (see Section 3.6 of | A ARj ) . this derivation 
is due to Koornwinder. [15] . Recall that its is given by 



1 

27 



ntiTK 



r(2is) 



ds = 



riKfc<is:4 r ( Q fc + aQ 
r(dl + a 2 + a 3 + a 4 ) 



(2.9) 



b) Another beta-integral. Applying the Plancherel formula to a; u °, x 
we get the integral 



1 

2^ 



nLi r K + is) 



T{2is)T{b- 



ds 



Tip - ai - a 2 - n 3 ) IIi<fc<;^3 T ( a k + a 

nLi r ( fo - a fe) 
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c) An integral representation for 3-^2(1)- The pair of functions 

a + c,a + d 



(l + x)- a - 



and 



2*1 



a + b + c + d 1 



gives the following integral representation of 3^2(1), 



r(o + is)T(b + is)T(c + is)T(d + is)T(e + is) 



T(2is) 



ds 



T(a + b)T{a + c)T(a + d)T(a + e)T(b + c)T(b + d)T(b + e)T{c + d)T(c + e) 



T(a + b + c + d)T(a + b + c + e) 

a + c, b + c. a + b 



x 3F2 



a + b + c + d, a + b + c + e 1 



(2.10) 



The left hand side is symmetric with respect to the parameters, therefore the 
right hand side also is symmetric. This symmetry is the Kummcr identity (sec 
[ME] , Corollary 3.3.5). 

d) Adding a T-factor to the numerator. Applying the Plancherel formula to 
the pair 



2F1 



a + c, a + d 



a + b + c + d' 
wc get the identity 



and 



2F1 



a + e,a + f _ 
a+b+e+ f 



1 

TT Jo 



T(a + is)T(b + is)T(c + is)T{d + is)T(e + is)T(f + is) 



r(2£s) 



ds 



2iri 

x — / r 

2ni 



T r(a + c)T(a + d)T(c + d)T{b + e)T(b + f)T(e + /) x 



a + b + s, a + e + s, a + / + s, d — a — s, c — a — s, ~s 
c + d— s,a + b + e + f + s 



ds. 

(2.11) 



The right hand side is a linear combination of three functions 4-^3(1) with T- 
coefheients. By the way a Barnes integral can be regarded as a final answer. 

e) Adding a T-factor to the denominator. Now we apply the Plancherel 
formula to the pair 



2^1 



p + b, q + b 



(1 + x) 



b—a 



and 2F1 



u + b.v + b 



(1 + x) 



b—a 
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We omit intermediate calculations and present the final result 



T(b + is)T(p + is)T(q + is)T(u + is)T(v + is) 



T(2is)T(a + is) 



ds 



2m 



u + v,p + q,p + b,q + b 
a — v.u — v 



rioo 

x / r 

-200 



u + p + s, u + q + s, b + u + s, a — v + s, v — u — s, — s 
u + a + s,u + b + p + q + s 



(2.12) 



Two last identities are not as aesthetic as previous. However, consider two 
special cases of the last integral. 

f ) The Nassrallah-Rahman integral. In the last integral we set a = b + u + 
v + p + q (this leads to cancellation of T-factors) and applying Theorem 2.4.3 
of |AAR| (after changing notation) , we obtain the Nassrallah-Rahman integral 
(its g-version is present in the book AAR , Theorem 10.8.2) 



1 

2^ 



n =1 r(a; 



T(2is)T(J2l a, + is) 



ds = 2- 



Ili<fc</<5 r(a fe + a ; ) 



Ilfc=l r ( a l + a 2 + a 3 + a 4 + a 5 - Ofc) 



g) The Whipple identity and the symmetry of the Wilson polynomial^ with 
respect to the parameters. We represent the right hand side of (|2.12[) in terms 

Of 4^3, 



u + v,p + q,p + b, q + b 
a — v,u — v 

v — u,u+p,u + q,u + b,a — v 
u + a,u + b + p + q 



+r 



u — v,p + v,q + v,b + v,a — u 
v + a,v + b + p + q 



4F3 



u + p, u + q, u + b, a — v 
l + u — v,u + a,u + b + p + q' 

p + v, q + v,b + v, a — u 
1 — v + u,v + a,v + b + p + q' 



(2.13) 



The left hand side of (12.12)) is symmetric with respect to the parameters b, p, 
q, u, v, but the right hand side in the form (|2.13[) does not looks as symmetric. 
This gives a symmetry relation for 4F3 in the form "a linear combination of four 
summands equals 0" . This is the " nonterminating Whipple identity" . Its " ter- 
minating version" (Theorem 3.3.3 of | A ARj ) can obtained by the substitution 



2 Recall that the Wilson polynomials W n (a, b, c, d; s 2 ) are given by the formula 

-n, n + a + b + c + d— I, a + is, a — is 



W n (a,b,c,d;s' ! ) = (a + b) n (a + c) n (a + d) n 4F3 



They are orthogonal with respect to the weight (see l|2.9l 



a - 



,a + c,a- 



w(s) = — 



F(a + is)T(b + is)T(c + is)T(d + is) 
T(2is) 
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a = v — m with integer m, then two summands disappear due factors T(—m) 
in the denominators (this identity is the symmetry of the Wilson polynomials 
with respect to the parameters). 

h) One again extension of the De Branges- Wilson integral. Applying the 
Plancherel formula to (I + x) b - a (l + x + y)- b - c and {1 + x) b - a (l + x + y)- b - d , 
we get 



r(o + is)T(b + is)T(c + is)T{d + is) 



X 2^1 



T(2is) 
c — is.c + is 



1F1 



d — is,d + is 



a + d 



ds = (2.14) 



7iT(a + b)T{a + c)T(a + d)T(b + c)T(b + d)T{c + d) 
T(a + b + c + d) 



2F1 



2b + c + d, c + 
a + b + c + d 



;-y 



2.4. A derivation of the orthogonality relations for Wilson poly- 
nomialsJl Now we derive J^-image of the function 

\T(a+is)\ 2 W n (s 2 ), where W n {s 2 ) = W n (a,b,c,d;s 2 ) is a Wilson polynomial. 
We must evaluate the integral 



T(a + is)T(b + is)T(c + is) 



T(b + c)J T(2is) 
_ (a + b) n (a + c) n (a + d) n 



1F1 



b + is,b — is 



T{b + c) 



b + c ' 
r(a + is)r(b + is)T(c + is) 



W n {s 2 )ds 



T(2is) 



x 2 F 1 



b + is,b — is 
b + c ' 



E 

fe=0 



(—n)k(n + a + b + c + d — l)k(a + is) k (a — is) k 
k\{a + b)k(a + c)u{a + d)k 



ds. 



We get a linear combination of known for us (in virtue of the inversion formula 
and f|2 .3|) ) integrals of the type 



T(a + k + is)T(b + is)T(c + is) 



T(2is) 



2^1 



, b — it 



ds — 



As a result we get 

T(a + b)T(a + c)(a + b) r 
ftp + c) 



(l + x) 



-a — b tj 
2-fl 



b + c ' 
_ T(a + b + k)T(a + c + k) 
~ (1 + x) a+b 

-n, n + a + b + c + d— 1 1 



a + d 



l + x 



This is a Jacobi polynomial of the variable 1/(1 + x). 



3 Another simple derivation is contained in 1231 . 
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Transposing a c d, we evaluate the inverse index transform of \T(d+is)\ 2 W m (s 2 ) 
Next we evaluate integral (see formula (3.8.3) of |AARj ) 



r(& + c) Jo 



T(a + is)T(b + is)T{c + is)T(d + is) 



r(2is) 



W n {s 2 )W m (s 2 )ds 



applying the Plancherel formula. We get 



const 



2 -Pi 
X 2^1 



1 



-n, n + a + b + c + d — 1 

a + d ' 1 + x 

-to, m + a + b + c + d — 1 1 



B 6+c - 1 (l + a;)- a -' , - c da;. 



Passing to the variable y = 1/(1 + x), we get the integral, which express the 
orthogonality relations for the Jacobi polynomials. 

At the first glance this proof of orthogonality can be invented only if we 
know the final result. Really this calculation gives us the following observation: 

The Wilson orthogonal system with a — d is the image of the Jacobi system 
under the index transform f[15|). 

Notice that the index transform was discovered in 1910 and became well- 
known upto 1950, therefore it seems strange that the Wilson polynomials were 
discovered so late (1980). 



3 Derivation of the inversion formula. Jump of 
resolvent 

Many ways of derivations are known, see [14]. In particular, we can decompose 
the index transform as a products of simpler integral transforms and apply 
inversion formulas for the factors. However, the original way of Weyl based on 
spectral theory seems the most natural up to now (see, e.g., [3], §13.8, or [31]). 
We present a version of derivation using a minimum of theory but requiring 
superfluous calculations. In details, the spectral theory of differential operators 
is exposed in Titchmarsh [3T], Dunford, Schwartz [3J, Chapters XII-XIII, and 
Naimark |21) . 

3.1. Jump of the resolvent. Recall the Spectral Theorem. Consider a 
finite or countable collection of measures Hi, /i 2 ,...on 1, the Hilbert space 

V[fi :=® j L 2 (R, t x j ), 

and an operator Zp : V[fl] — > V[p] given by the formula 

© h e ■ • •)] (s) = xf x {x) ® xf 2 {x) © . . . 

Theorem 3.1 For any self-adjoint (generally, unbounded) operator in a Hilbert 
space H there exists a collection of \ij and a unitary operator U : H — > V[p\ 
such that A = U~ 1 ZplJ 
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For any Borel subset Af C I consider the subspace W(M) c V\p\ of 
functions, which equals outside the set M. Define the spectral subspace 
fi(M) := U~ 1 W(M). Denote by P[Sl] the projection operator to this subspace. 

Proposition 3.2 For any finite interval (a, b) C M, 

1 r b ~ s 

P\(a,b)]= lim lim / ((A - ie - AY 1 - (A + ie - A)- 1 ) dX. 

LV n 2ni 6^+Oe^+oJ a+5 yy 1 

The limit here is the limit in the strong operator topology, T n —¥ T if for 
any vector v we have \\T n v — Tv\\ —> 0. 

A verification of this statement is straighforwarward (and is a good exercise, 
in particular for finite-dimensional spaces), we can from outset assume that our 
operator acts in V\p\. 

For any vector v, 

1 f N 
v = lim lim / ((A - ie — A) 1 - (A + ie - A) l )vdX. 

2lTl iV-)-oo 7-JV 

An evaluation of the limit gives the spectral decomposition. Now we will 
perform this for the hypergeometric differential operator D defined above (|1.5p . 

3.2. Solutions of the equation (D — X)f = 0. For each A this equation 
has two linear independent solutions. We choose two bases in the space of 
solutions (both bases consist of Kummer series, see [4], Section 2.9). The first 
basis consists of functions 

ip(x,X)= 2 F 1 [b + VX 1 b-v / X;b + c:-x}; (3.1) 

ip(x,X) = (~x) 1 - b - c 2 F 1 [l + VX-c, 1 - VX-c;2-b-c;-x]. (3.2) 
The second basis u±(x) is given by formulas 

«±(x,A) = (~x)- bTVX 2 F 1 [b±VX,l±VX-c;l±2VX;~ X - 1 }. (3.3) 

We assume that the complex plane A is cut along the negative semi-axis. 

For the first pair of functions the behavior near zero is easily observable, 
for the second pair we see the behavior near infinity. Below we need a formula 
expressing ip in terms of m+ and 

<p(x,\) = B + (X)u + (x,X) + 5_(A)u_(x,A), 

where 

b ± (x)= r(fc + c)r( T VA) 

T(b T VX)T(cTVX) 

3.3. Self-adjointness. Let b > 0, c > 0. We define the operator D on the 
space D(R + ) of smooth compactly supported functions on (0, oo). The operator 
D is formally symmetric with respect to the weight x h+c_1 (l + x) b ~ c dx, i.e., 



(Df)(x)g(x)x b+c -\l + x) b - c dx = / f{x)Dg(x)x b+c -\l + xf' dx 
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where f,g£ D(R+). Its adjoint operator D* is determined from the condition 
D*g = hifg,h£ L 2 (R+,x b+c - 1 (l + xf- c ) and 



(Df)(x)g(x)x b+c - 1 (l + x) b - c dx = / f(x)h{x)x b+c - 1 (l + x) b ~ c dx 
o Jo 

for all / G D(M+). As before, this operator is given by formula (|1.5I) . but its 
domain of definition have increased. 

Recall that for any formally symmetric operator A the numbers dim ker(/l* — 
A) (deficiency indexes) are constant on the half-planes ImA > and ImA < 0. 
The operator A is essentially self-adjoint if the both numbers are 0. Therefore 
we must verify existence/nonexistence of solutions of the differential equations 
Df = Xf with ImA =^ such that / is contained in L 2 with respect to our 
weight. To be definite consider the upper half-plane ImA > 0. 

It is easy to see that for b + c > 2 such solutions do not exist. Indeed, tp is 
too large near 0, and U- is too large near oo. Therefore an L 2 -solution must 
coincide with tp and u+ simultaneously. But these two solutions arc different. 

Therefore D is essentially self-adjoint. 

Remark. If b + c < 2, then tp and tp are in L 2 near 0. Therefore u + € L 2 
and the operator D is not self-adjoint. We extend the operator D and define 
it on the space of functions smooth on the closed half-line [0, oo) and vanishing 
for large x. Then the operator became self-adjoint. Below we do not watch this 
case. □ 

3.4. The resolvent. 

Lemma 3.3 The resolvent (D — A) -1 of the operator D is defined in the domain 
C \ [— oo, 0) and is given by 

/•OO 

L(X)f(x)= K(x,y;X)y b+c - 1 (l + y) b+c dy 7 (3.5) 
Jo 

where the 'Green function' K is given by 

K( [2B_{X)-'X- l / 2 V {x,X)u + { yi Xl ^fx^y 

K{X ^ X) -\2B_{X)^X-VMy,X)u + {x,Xl if x > y ' ^ 

B-(X) is defined by {33]). 

The jump of the resolvent appear on the semi-axis A ^ due discontinuity 
of \/A on the cut. Evaluating the jump of resolvent we get 

Remark. Formula is so simple, because tp has no jump; jump of u+ is 
proportional to ip. □ 

The last formula is the desired inversion formula. 
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Proof of Lemma. First, we formally check the identity (D — A)L(A) = 1. 
We must verify that the function K satisfies the equation 



(D x -X)K(x,y ] X)y t>+c - 1 (l+y) 



\b+c 



5(x - y). 



(3.8) 



Obviously, outside the diagonal x = y the equality (D x — X)K = holds. The 
kernel K is continuous, but the first derivative has a jump. Therefore, 



{D x -X)K(x,y;X) = x(x+l){ 



dK(x, y, A) 



dx 



dK(x, y, A) 



y—x+Q 



dx 



\5{x-y) 

y—x—0 ) 



= 2S_(A)- 1 A- 1 /2 \y u+ (y, A) - <p(y, X)u + (y, A)'] S(x - y). 

In square brackets we have Wronski determinant of two solutions of {D — A)/. 
Upto a constant factor Wronskian is detemined by a differential equation, in 
our case it equals const • y~ b ~ c (l + y) c ~ 6-1 . To evalute the constant factor we 
watch asymptotics of the Wronskian as y — > oo. 

In fact this calculation is sufficient to a proof. But a priory boundedness of 
-L(A) in L 2 is not evident. We overcome this difficulty in the following way. 

Since D is essentially self-adjoint, for A ^ R the operator (D — A) -1 is 
unbounded. In virtue of L. Schwartz's Kernel Theorem (see, e.g., (. [10] ) 
(D — A) -1 is an integral operator, its kernel K(x, y; A) is a distribution of two 
variables. It satisfies equation (|3.8p and the symmetry condition K(y, x; X) — 
K(x,y,X). Therefore, outside the diagonal x = y the distribution satisfies the 
system of equations 

(D x - X)K = 0, (D y - X)K = 0. 

It can be readily checked that our kernel K is a unique admissible candidate, 
all other solutions of the system increase too rapidly. □ 

3.5. The Romanovski Polynomials. Now let b < 0, b + c > 0. Let 

m = 0, 1, . . . , [— b]. Consider the polynomials p m given by 



p m {x) 



2 -Pi 



-m, 2b + r7i 
b + c ' 



x m T(b + c)r(-m - b) 
T(2b + m)T(c + b + m) 



2 r l 



-777, 1 — 777 — 5 

2-b-c 



Theorem 3.4 a) The polynomials p m are contained in L 2 (R+ , x fc+c 1 (l+x) b c ). 

b) Dp m = (b + m) 2 p m . 

c) The polynomials p m are pairwise orthogonal. 

The statements a), b) are evident, ) follows from a) and b). □ 

Thus we get a finite system of orthogonal polynomials. We can not enlarge 
it because the monomials a;^ with larger powers are not in I? . 
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Remark. A lot of such finite systems of orthogonal polynomials is known. 
Romanovski [29] introduced another two systems: the polynomials on the line 
orthogonal with respect to the weight 



dx 



where a £ C, 



(1 + ix) a (l - ix) 



I a 



(they also are analytic continuations of the Jacobi polynomials) and polynomials 
on the half-line x ^ with respect to the weight 



(this is an analytic continuation of the Laguerre polynomials of 1 / x) . More com- 
plicated finite systems of orthogonal polynomials were enumerated by P. Lesky 



Our considerations explain this phenomenon. For 6 < 0, b + c> our opera- 
tor D has finite number of discrete eigenvalues corresponding to the Romanovski 
polynomials, which are added to the continuous spectrum. 

It is necessary to modify our calculation. The resolvent (see formula (I3.6[) ) 
now has a finite number of poles at points A = (b+m) 2 , they arise from the poles 
of £>_(A) _1 . To write the jump of the resolvent we must additionally evaluate 
residues at these poles. In the inversion formula (|3.7p we get additional terms 



The expression ()1.3[) for Jb :C does not change. But the function J& iC /(s) 
now is defined on the following subset in C: the half-line s ^ and a finite set 
s = i(b + m) on the imaginary axis (these points correspond to Romanovski 
polynomials). 

Remark. Such orthogonal systems arise in non-commutative harmonic 
analysis and correspond to discrete part of spectra (for instance, for L 2 on 
pseudo-Riemannian symmetric spaces of rank 1, see also |24] about tensor 
products of unitary representations of the group SL(2,R). Apparently (no- 
body verified this) the discrete Flensted- Jensen series [5] are controlled by some 
multivariable orthogonal systems of Romanovski type. 

4 Applications to harmonic analysis 

4.1. Pseudounitary groups of rank 1. Let IK be R, C or the quaternion 
algebra H. The case IK = R is sufficiently interesting. We present several simple 
facts without proofs, the reader can believe or verify. 

Denote by r the dimension of IK. Let K™ be the n-dimensional space over IK 
with the standard inner product, 



x ^exp(—l/x)dx 



[T5] , [T7] . some additions are in |23|). 



□ 
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By U(l,n;K) we denote the pseudounitary group over K, i.e., the group of 
(1 + n) x (1 + ra)-matrices ( " J) over K satisfying the condition 

fa b\ f-l (A fa bY f-l 
\c d) \ l) \c d) ~ \ 1 

The standard notations for the groups U(l,n;K) for K = K,C,H are respec- 
tively: 0(1, n), U(l,n), Sp(l,n). 

4.2. Homogeneous hyperbolic spaces. Denote by B„(K) the open unit 
ball (z,z) < 1 in IK™. By 5 rn_1 we denote the sphere (z,z) — 1. The group 
U(l,n;K) acts on B„(K) by linear-fractional transformations 

z H- z [9] := (a + zc)-\b + zd). (4.1) 

The stabilizer K of the point € B„(K) consists of matrices of the form 

|a| = l, deU(n;K). (4.2) 

Therefore B„(K) is the homogeneous space 

B n (K) = U(l, n; K) /(U(l; IK) x U(n; K)). 

Remark. If K = R, then our ball is the n-dimensional Lobachevsky space 
in the Beltrami-Klein model. Recall that in this case straight lines in the 
Lobachevsky sense are segments (chords), the sphere S*™ -1 is the absolute in 
the Lobachevsky sense. The group 0(1, n) is the group of motions of the 
Lobachevsky space. 

For IK = C and H we get complex and quaternionic hyperbolic spaces. □ 

The Jacobian of the transformation (1.5) is 

J(g;z) = \a + zc\- r{1+n) . 

Note the following simple formula 

1 - (z l9 \u l9] ) = (a + zc)- 1 ^ - { Zl u)){a + uc)- 1 . 

This implies than the U(l, n; K)-invariant measure on B n (K) has the form 

dm(z) = (l-(z,z))- (n+1 >^dz, 

where dz denotes the Lebesgue measure on B„(K). 

The group U(l,n;K) acts in L 2 (B„ (IK), dm(z)) by changes of variable 

p(g)f(z) = f{(a + zc)- 1 (b + zd)). (4.3) 

Evidently these operators are unitary. In other words we get an infinite dimen- 
sional unitary representation of the group U(l,n, K). 
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Our next problem is to decompose this representation into irreducible repre- 
sentations. 

4.3. The spherical principal series. Let s E M. A representation T s of 
spherical principal series of the group U(l, n; K) is realized in L 2 (5 ,rn_1 ) and is 
given by the formula 



(l b ^f(h) = f{(a + hc)- 1 (b + hd))\a + hc\-( n + i y^ 2+1 +™, (4.4) 



where h £ S rn 1 . A straightforward calculation shows that these representa- 
tions are unitary for set. 

Remark. All representations T s are irreducible, representations T s and T_ s 
are equivalent (this is not completely obvious, see [3"2"]). 

Remark. The term 'series' is used because these groups have different types 
of unitary representations. The term 'spherical' means that any representation 
T s contains a (unique) i^-invariant vector. In our model this vector is the 
function / = 1. 

4.4. An intertwining operator. Consider the space of functions ip(h, s) 
on the semi-cylinder S™' 1 x R + (a precise description of this space is given 
below), let U(l,n;K) act in this space by the formula 

r r b A (p(h, s) = <p((a + hc)- 1 ^ + hd), s) \a + hc \-(n+l)r/2+l+is_ 

For a fixed s we get the representation T s {g) in functions depending on h. Thus 
we have some kind of a direct sum of all representations T s with respect to a 
continuous parameter s (thus is called a 'direct integral'). 

We define the following operator A from the space L 2 (B n (K), dm(z)) to the 
space of functions on 5' r ™ _1 x R + : 

f |1 - (z M|-("+l)'-/2+l+M 

Af{k ' S) = 1b„ (K) / W|l-( Z> ;>|(n + lW4 + l/2 + W2 dZ - ^ 

Lemma 4.1 The operator A is intertwining, i.e., 

A p(g) = T (g) A , f or al1 9- 

This statement is a useful two-step exercise. First, it is worth to verify the 
lemma in a straightforward way. Secondly, it is interesting to find a way to 
invent the formula for the operator A if you do not know it before. 

4.5. The Plancherel formula. 



Theorem 4.2 The operator A is a unitary operator 



i 2 (B„(K),dm(z)) -> L 2 ^S*™" 1 x 
where 



T(b + is)T(c + is) 2 



dsdh), (4.6) 



r(2is) 

(n + l)r/4-l/2; c = (n - l)r/4 + 1/2. (4.7) 
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Keeping in the mind the previous lemma we get that the operator A identifies 
the representation of the group U(l, n; K) in L 2 on the ball with a continuous 
direct sum of representations of principal series. 

Beginning of proof. First, we explain the appearance of the T-factor. 
For this purpose we restrict the operator A to the space of functions depending 
only on radius. It is convenient to define the variable 

x - H 2 
i-N 2 

and set / = f(x). Then the corresponding function G(h,s) depends only the 
variable s, an uncomplicated calculation gives the familiar formula 

p oo 

G{s) = const • / f(x) 2 Fi{b + is,b- is; b + c; -x)x b+c - 1 (l + x) b - c dx. (4.8) 
Jo 

Thus, we observe that the operator A is a unitary operator from the space 
of i 2 -functions on the ball depending only on radius to the space of functions 
on the half-cylinder depending only on s. 

This is the main argument], it remains to apply some standard representation- 
theoretic tricks. 

4.6. The end of the proof. Denote G := U(l,n;K), K := U(1,K) x 
U(n, IK). Denote the Hilbert spaces L 2 from row (j4.6[) by V and W respectively. 
By V K and W K we denote the spaces of if-fixed vectors in V and W. By Py 
and Pw we denote the projection operators to V K and W K ■ 

Recall the following standard statement. 

Lemma 4.3 Let p(k) be a unitary representation of a compact group K . Then 
the projection operator to the space of K -fixed vectors is given by the formula 

P = I p(k)dk 7 
Jk 

where dk is the Haar measure on K , normed in such a way that the measure of 
the whole group is 1. 

Corollary 4.4 P W A = AP V . 

Lemma 4.5 Any closed G-invariant subspace in V contains a smooth function. 

Proof. Consider a sequence of smooth compactly supported positive func- 
tions rj on G approximating the (5-function at unit. For a vector v ^= from the 
subspace we get a sequence of smooth functions J rj(g)p(g)v dg, convergent to 
v. □ 

4 For K = C there are also actions of the group U(l,n) given by 

pGjO/O*) = /((« + zc)~ l {b + zd)) (a + zc) k (a + zc) ~ k 

The problem of decomposition also is reduced to the index transform (with another parame- 
ters), the representation has finite discrete spectrum which is controlled by the Romanovsky 
polynomials. 
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Lemma 4.6 Any closed G-invariant subspace in V contains a K -invariant vec- 
tor. 

Proof. Consider a smooth function / from the subspace. Let f(a) 7^ 0. 
Consider g 6 G such that 0^ — a. Next, average the function f(x^) by K . □ 

Corollary 4.7 The linear span of vectors p{g)v, where g ranges in G and v 
ranges in V K , is dense in V. 

Proof. If not, we consider the orthogonal comlement to this kinear span. 
It contains a if- invariant vector. □ 

Lemma 4.8 The linear span of vectors r{g)w, where g ranges in G and w 
ranges in W , is dense in W . 

Proof. We use irreducibility of representations T s of the principal series. 
As w we take functions f(x, s) = 1, if \s — sq\ < e and otherwise. It is easy to 
show that there are no functions orthogonal to all r g (g)f . □ 

End of proof of Theorem 14.21 Let p(g)v, p(g')v' be as in the last 
lemma. Then 

/\ f\ (representation U is unitary) , . _-i f f . (Py is projection operator) 



{p(g)v,p{g)v)v = {v,p(g g)v) v 

1 r, 1 -1 l\ l\ (Planchcrcl formula) . . _i 

= (v,P v p(g V)uV = (Av,AP w p(g 1 g')v') w = 

(A is intertwining) , . f _ i », . ». (Pw ls a projection operator) 

(Av,P W T[g g)Av) w 



, . , 1 , . /. (representation r is unitary) , / s , / /x . /, 

= {Av,r(g- 1 g)Av') w " P - (r(g)Av, r{g')Av') w . 

Therefore A is an isometry. On the other hand, the image of A contains W K 
(by Theorem I Lip and therefore contains the whole W (by Lemma 
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